{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 4\n",
    "\n",
    "Regression\n",
    "\n",
    "Allen Downey\n",
    "\n",
    "[MIT License](https://en.wikipedia.org/wiki/MIT_License)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set(style='white')\n",
    "\n",
    "from utils import decorate\n",
    "from thinkstats2 import Pmf, Cdf\n",
    "\n",
    "import thinkstats2\n",
    "import thinkplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple regression\n",
    "\n",
    "An important thing to remember about regression is that it is not symmetric; that is, the regression of A onto B is not the same as the regression of B onto A.\n",
    "\n",
    "To demonstrate, I'll load data from the BRFSS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time brfss = pd.read_hdf('brfss.hdf5', 'brfss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brfss.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few people report many vegetable servings per day.  To simplify the visualization, I'm going to replace values greater than 8 with 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = brfss['_VEGESU1'] > 8\n",
    "brfss.loc[rows, '_VEGESU1'] = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use SciPy to compute servings of vegetables as a function of income class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import linregress\n",
    "\n",
    "subset = brfss.dropna(subset=['INCOME2', '_VEGESU1'])\n",
    "xs = subset['INCOME2']\n",
    "ys = subset['_VEGESU1']\n",
    "\n",
    "res = linregress(xs, ys)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Increasing income class by 1 is associated with an increase of 0.07 vegetables per day.\n",
    "\n",
    "So if we hypothesize that people with higher incomes eat more vegetables, this result would not get us too excited.\n",
    "\n",
    "We can see what the regression looks like by plotting the line of best fit on top of the scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_jitter = xs + np.random.normal(0, 0.15, len(xs))\n",
    "plt.plot(x_jitter, ys, 'o', markersize=1, alpha=0.02)\n",
    "\n",
    "plt.xlabel('Income code')\n",
    "plt.ylabel('Vegetable servings per day')\n",
    "\n",
    "fx1 = np.array([xs.min(), xs.max()])\n",
    "fy1 = res.intercept + res.slope * fx1\n",
    "\n",
    "plt.plot(fx1, fy1, '-', color='C1');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do it the other way around, regressing income as a function of vegetable servings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = subset['_VEGESU1']\n",
    "ys = subset['INCOME2']\n",
    "\n",
    "res = linregress(xs, ys)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we can plot the line of best fit on top of the scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_jitter = ys + np.random.normal(0, 0.3, len(xs))\n",
    "plt.plot(xs, y_jitter, 'o', markersize=1, alpha=0.02)\n",
    "\n",
    "plt.ylabel('Income code')\n",
    "plt.xlabel('Vegetable servings per day')\n",
    "\n",
    "fx2 = np.array([xs.min(), xs.max()])\n",
    "fy2 = res.intercept + res.slope * fx2\n",
    "\n",
    "plt.plot(fx2, fy2, '-', color='C2');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The slope looks more impressive now.  Each additional serving corresponds to 0.24 income codes, and each income code is several thousand dollars.  So a result that seemed unimpressive in one direction seems more intruiging in the other direction.\n",
    "\n",
    "\n",
    "But the primary point here is that regression is not symmetric.  To see it more clearly, I'll plot both regression lines on top of the scatter plot.\n",
    "\n",
    "The green line is income as a function of vegetables; the orange line is vegetables as a function of income.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_jitter = ys + np.random.normal(0, 0.3, len(xs))\n",
    "plt.plot(xs, y_jitter, 'o', markersize=1, alpha=0.02)\n",
    "\n",
    "plt.ylabel('Income code')\n",
    "plt.xlabel('Vegetable servings per day')\n",
    "\n",
    "fx2 = np.array([xs.min(), xs.max()])\n",
    "fy2 = res.intercept + res.slope * fx2\n",
    "\n",
    "plt.plot(fx2, fy2, '-', color='C2')\n",
    "plt.plot(fy1, fx1, '-', color='C1');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the same thing the other way around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = subset['INCOME2']\n",
    "ys = subset['_VEGESU1']\n",
    "\n",
    "res = linregress(xs, ys)\n",
    "res\n",
    "\n",
    "x_jitter = xs + np.random.normal(0, 0.15, len(xs))\n",
    "plt.plot(x_jitter, ys, 'o', markersize=1, alpha=0.02)\n",
    "\n",
    "plt.xlabel('Income code')\n",
    "plt.ylabel('Vegetable servings per day')\n",
    "\n",
    "fx1 = np.array([xs.min(), xs.max()])\n",
    "fy1 = res.intercept + res.slope * fx1\n",
    "\n",
    "plt.plot(fx1, fy1, '-', color='C1')\n",
    "plt.plot(fy2, fx1, '-', color='C2');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StatsModels\n",
    "\n",
    "So far we have used `scipy.linregress` to run simple regression.  Sadly, that function doesn't do multiple regression, so we have to switch to a new library, StatsModels.\n",
    "\n",
    "Here's the same example from the previous section, using StatsModels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.formula.api as smf\n",
    "\n",
    "model = smf.ols('INCOME2 ~ _VEGESU1', data=brfss)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is an `OLS` object, which we have to `fit`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.fit()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`results` contains a lot of information about the regression, which we can view using `summary`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the parts we're interested in is `params`, which is a Pandas Series containing the estimated parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `rsquared` contains the coefficient of determination, $R^2$, which is pretty small in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can confirm that $R^2 = \\rho^2$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sqrt(results.rsquared)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ['INCOME2', '_VEGESU1']\n",
    "brfss[columns].corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Run this regression in the other direction and confirm that you get the same estimated slope we got from `linregress`.  Also confirm that $R^2$ is the same in either direction (which we know because correlation is the same in either direction)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple regression\n",
    "\n",
    "For experiments with multiple regression, let's load the GSS data again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time gss = pd.read_hdf('gss.hdf5', 'gss')\n",
    "gss.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore the relationship between income and education, starting with simple regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = smf.ols('realinc ~ educ', data=gss)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.fit()\n",
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like people with more education have higher incomes, about $3586 per additional year of education.\n",
    "\n",
    "Now that we are using StatsModels, it is easy to add explanatory variables.  For example, we can add `age` to the model like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = smf.ols('realinc ~ educ + age', data=gss)\n",
    "results = model.fit()\n",
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the effect of `age` is small, and adding it to the model has only a small effect on the estimated parameter for education.\n",
    "\n",
    "But it's possible we are getting fooled by a nonlinear relationship.  To see what the age effect looks like, I'll group by age and plot the mean income in each age group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = gss.groupby('age')\n",
    "grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_income_by_age = grouped['realinc'].mean()\n",
    "\n",
    "plt.plot(mean_income_by_age, 'o', alpha=0.5)\n",
    "plt.xlabel('Age (years)')\n",
    "plt.ylabel('Income (1986 $)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yeah, that looks like a nonlinear effect.\n",
    "\n",
    "We can model it by adding a quadratic term to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['age2'] = gss['age']**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = smf.ols('realinc ~ educ + age + age2', data=gss)\n",
    "results = model.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the coefficient associated with `age` is substantially larger.  And the coefficient of the quadratic term is negative, which is  consistent with the observation that the relationship has downward curvature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** To see what the relationship between income and education looks like, group the dataset by `educ` and plot mean income at each education level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Maybe the relationship with education is nonlinear, too.  Add a quadratic term for `educ` to the model and summarize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['educ2'] = gss['educ']**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = smf.ols('realinc ~ educ + educ2 + age + age2', data=gss)\n",
    "results = model.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameters of a non-linear model can be hard to interpret, but maybe we don't have to.  Sometimes it is easier to judge a model by its predictions rather than its parameters.\n",
    "\n",
    "The results object provides a `predict` method that takes a `DataFrame` and uses the model to generate a prediction for each row.  Here's how we can create the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame()\n",
    "df['age'] = np.linspace(18, 85)\n",
    "df['age2'] = df['age']**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`age` contains equally-spaced points from 18 to 85, and `age2` contains those values squared.\n",
    "\n",
    "Now we can set `educ` to 12 years of education and generate predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(mean_income_by_age, 'o', alpha=0.5)\n",
    "\n",
    "df['educ'] = 12\n",
    "df['educ2'] = df['educ']**2\n",
    "pred12 = results.predict(df)\n",
    "plt.plot(df['age'], pred12, label='High school')\n",
    "\n",
    "plt.xlabel('Age (years)')\n",
    "plt.ylabel('Income (1986 $)')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot shows the structure of the model, which is a parabola.  We also plot the data as an average in each age group.\n",
    "\n",
    "**Exercise:**  Generate the same plot, but show predictions for three levels of education: 12, 14, and 16 years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding categorical variables\n",
    "\n",
    "In a formula string, we can use `C()` to indicate that a variable should be treated as categorical.  For example, the following model contains `sex` as a categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "formula = 'realinc ~ educ + educ2 + age + age2 + C(sex)'\n",
    "results = smf.ols(formula, data=gss).fit()\n",
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimated parameter indicates that `sex=2`, which indicates women, is associated with about \\$4150 lower income, after controlling for age and education.\n",
    "\n",
    "**Exercise**: Use `groupby` to group respondents by `educ`, then plot mean `realinc` for each education level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Make a `DataFrame` with a range of values for `educ` and constant `age=30`.  Compute `age2` and `educ2` accordingly.\n",
    "\n",
    "Use this `DataFrame` to generate predictions for each level of education, holding age constant.  Generate and plot separate predictions for men and women.\n",
    "\n",
    "Also plot the data for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression\n",
    "\n",
    "Let's use logistic regression to see what factors are associated with support for gun control.  The variable we'll use is `gunlaw`, which represents the response to this question: \"Would you favor or oppose a law which would require a person to obtain a police permit before he or she could buy a gun?\"\n",
    "\n",
    "Here are the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['gunlaw'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 means yes, 2 means no, 0 means the question wasn't asked; 8 and 9 mean the respondent doesn't know or refused to answer.\n",
    "\n",
    "First I'll replace 0, 8, and 9 with NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['gunlaw'].replace([0, 8, 9], np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to put `gunlaw` on the left side of a regression, we have to recode it so 0 means no and 1 means yes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['gunlaw'].replace(2, 0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what it looks like after recoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss['gunlaw'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run a logistic regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = smf.logit('gunlaw ~ age + age2 + educ + educ2 + C(sex)', data=gss).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the parameters.  The coefficient of `sex=2` is positive, which indicates that women are more likely to support gun control, at least for this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other parameters are not easy to interpret, but again we can use the regression results to generate predictions, which makes it possible to visualize the model.\n",
    "\n",
    "I'll make a `DataFrame` with a range of ages and a fixed level of education, and generate predictions for men and women."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = gss.groupby('age')\n",
    "favor_by_age = grouped['gunlaw'].mean()\n",
    "plt.plot(favor_by_age, 'o', alpha=0.5)\n",
    "\n",
    "df = pd.DataFrame()\n",
    "df['age'] = np.linspace(18, 89)\n",
    "df['educ'] = 12\n",
    "\n",
    "df['age2'] = df['age']**2\n",
    "df['educ2'] = df['educ']**2\n",
    "\n",
    "df['sex'] = 1\n",
    "pred = results.predict(df)\n",
    "plt.plot(df['age'], pred, label='Male')\n",
    "\n",
    "df['sex'] = 2\n",
    "pred = results.predict(df)\n",
    "plt.plot(df['age'], pred, label='Female')\n",
    "\n",
    "plt.xlabel('Age')\n",
    "plt.ylabel('Probability of favoring gun law')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Over the range of ages, women are more likely to support gun control than men, by about 15 percentage points.\n",
    "\n",
    "**Exercise:** Generate a similar plot as a function of education, with constant `age=40`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Use the variable `grass` to explore support for legalizing marijuana.  This variable record the response to this question: \"Do you think the use of marijuana should be made legal or not?\"\n",
    "\n",
    "1. Recode `grass` for use with logistic regression.\n",
    "\n",
    "2. Run a regression model with age, education, and sex as explanatory variables.\n",
    "\n",
    "3. Use the model to generate predictions for a range of ages, with education held constant, and plot the predictions for men and women.  Also plot the mean level of support in each age group.\n",
    "\n",
    "4. Use the model to generate predictions for a range of education levels, with age held constant, and plot the predictions for men and women.  Also plot the mean level of support at each education level.\n",
    "\n",
    "Note: This last graph might not look like a parabola.  Why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
